System and method for generating a representative computerized display of a user&#39;s interactions with a touchscreen based hand held device on a gazed-at screen

ABSTRACT

A system and method of interacting with a smartphone application displayed over a remote display gazed at by a user including a normal mode of operation which may be the default, a hovering mode of operation which may be triggered by a trigger event, and a release event which reverts from hovering mode to normal mode. An enhanced hovering ability is typically provided while hovering, in that a predetermined set of operative gestures is enabled, such that a range of predetermined gestures is available for operating purposes while hovering.

REFERENCE TO CO-PENDING APPLICATIONS

Priority is claimed from U.S. provisional application No. 61/428,385,entitled “Method and system of providing a sensory cue for the positionof digits in a touch screen based interface” and filed 30 Dec. 2010.

FIELD OF THE INVENTION

The present invention relates generally to touch screen applications andmore particularly to touch screen applications which interact with anexternal display.

BACKGROUND OF THE INVENTION

Conventional technology pertaining to certain embodiments of the presentinvention is described in the following publications inter alia:

EP 199090707 describes a dual function touch screen for a portableconsumer electronics device, including computing a touch location signaland displaying a resulting cursor track signal.

U.S. Pat. No. 7,199,787 to Lee describes a touch screen and a method forgenerating a display via an external display device, includingrecognizing a touch pad location selected by a user and displaying aposition on the external display, accordingly.

U.S. Pat. No. 7,870,496 describes a touchscreen remotely controlling ahost computer in which the cursor is kept in the center of the displayexcept when a center of a selected portion of a host image is close toan edge of the host image.

The user interface of current generation of hand held consumerelectronic devices (such as smart phones and media players) is touchdriven. An emerging trend in this area is the remoting of the userinterface display of these devices onto external surfaces (either wallsor screens) thus allowing the user to hold a consumer electronics (CE)device at a distance from the actual display. This may be done by theuse of pico projectors, HDMI or DVI cables, or wireless video streamingtechnologies to stream the UI (user interface) of the CE device onto acomputer or a media streamer connected to a TV. In addition,combinations and variations of the above are also possible, such as aphysical HDMI connection to a third party device that transmits thevideo stream produced by the hand held device over a Wifi connection.

The user of the hand held device may interact with the device UI (userinterface) using the touch interface, by putting one or more fingers onthe touch screen and performing various gestures.

Existing technology includes using equipment either connected externallyto the CE device (e.g. external mouse, keyboard or joystick) or alreadyembedded in the CE device (such as DPAD or the SurePress technology ofBlackBerry® Storm™). These devices may provide an image (such as mousepointer) hovering above the place where the interaction may happen.Other instances of existing technology use depth sensing technologies inorder to detect a finger/s before they touch the display and provide avisual indication of the location of the touch.

The disclosures of all publications and patent documents mentioned inthe specification, and of the publications and patent documents citedtherein directly or indirectly, are hereby incorporated by reference.

SUMMARY OF THE INVENTION

Certain embodiments of the present invention seek to provide a systemand method of interacting with a smartphone application displayed over aremote display gazed at by a user including a normal mode of operationwhich may be the default, a hovering mode of operation which may betriggered by a trigger event, and a release event which reverts fromhovering mode to normal mode. An enhanced hovering ability is typicallyprovided while hovering, in that a predetermined set of operativegestures is enabled, such that a range of predetermined gestures isavailable for operating purposes while hovering.

Certain embodiments of the present invention pertain to user interactionwith an image displayed on a screen.

Certain embodiments of the present invention seek to resolve thisinconvenience to the user, enabling her or him to look only at theremote display, without the need to look at the touchscreen of the CEdevice.

Certain embodiments of the present invention seek to provide acompletely programmatic embodiment including a method and system whichuse only the multi touch sensing capability of the touchscreen embeddedin a CE device, without providing additional hardware and instead onlyaltering the software of the CE device.

According to certain embodiments of the present invention a hoveringmode is added to the normal operational, also termed herein “native” or“touchscreen” mode of the user interface of a handheld CE device whichis paired with an external, typically larger screen such as a TV orcomputer screen, such that the larger screen is displaying the imagereceived from the handheld CE device.

In a typical embodiment, the hovering mode is implemented as additionalcomputer software added to the conventional CE device code that handlesthe reading and handling of touch screen events from the hardware anddispatching it to higher levels of code in the CE device operatingsystem. In the hovering mode, the user moves one or more interactionelements, e.g. his own finger/s (similar to mouse pointer on a computerscreen). An ongoing sensory cue regarding the position of the element/se.g. finger/s is provided to the user such that s/he need not gaze atthe screen of the CE device. When the user wants to interactconventionally with the CE device (such as clicking a button, orperforming a gesture) the user so indicates, e.g. as described below,and then the hovering mode is interrupted to enable the user to interactconventionally with the CE device.

Certain embodiments of the present invention seek to provide a systemand method for entering the hovering mode from the CE device's normaltouchscreen mode, e.g. by effecting a trigger event.

Certain embodiments of the present invention seek to provide a systemand method for providing a sensory cue indicating the location of thepointing element/s e.g. finger/s while in hovering mode given that theuser moves the pointing element/s e.g. finger/s while in hovering mode.

Certain embodiments of the present invention seek to provide a systemand method for the user to interact with the underlying interface whilein hovering mode.

Certain embodiments of the present invention seek to provide a systemand method for native interface elements to know that the interactionelement/s e.g. finger/s is hovering above. For example, in an Androidsystem, the View that is focused at the location the finger is hoveringabove, may receive an onTouchEvent or onClickEvent call. According tocertain embodiments a passive interaction with the underlying interfacemay also be effected in hovering mode, such as when the visual cue (e.g.red circle) hovers above a particular native interface element. Forexample, in an Android system, the underlying view may receive aonHoverEvent if an appropriate Listener is registered with the Androidsystem.

Certain embodiments of the present invention seek to provide a systemand method for exiting from hovering mode back to the CE device's normaltouchscreen mode, e.g. by effecting a release event.

According to certain embodiments, entering the hovering mode includespressing a special button on the CE device (such as but not limited to along press on a volume key or camera key) and touching the touchscreenduring the following predetermined time period e.g. at least twoseconds.

According to certain embodiments, a visual cue may be provided e.g. inthe shape of a red circle in the last place the circle was when thehovering mode was exited, or in a predetermined location e.g. the centerof the screen if this is the first time the hovering mode is entered.

When a finger is moved on the touchscreen the visual cue may move Xinches for every Y pixels that the finger is moved on the screen.

Lifting the finger for a period of less than (say) 2 seconds may leavethe red circle (e.g.) in the same place, and the CE device remains inhovering mode. Placing the finger on the touchscreen again does notalter the visual cue's position.

According to certain embodiments, interacting with the underlyinginterface is triggered by performing some gesture with an additionalfinger. This gesture may for example include tapping, e.g. as shown inFIG. 3 b, at a designated location on the screen, typically in a mannerwhich is unique hence avoids conflict with conventional touch events, orsome other gesture may be performed such as swiping. The interactionwill be recorded at the location of the red circle, and not at thetapped location. Typically, the interaction at the tapped location willbe regarded as a finger touch on the touchscreen at the location of thered circle. For example, in Android systems the View that is focused inthe location of the red circle may receive an onTouchEvent oronClickEvent call.

Any suitable method may be employed to detect the tapping gesture, whichmay, for example, include some or all of the following operations:monitor entire touch screen, or only screen locations adjacent touchlocations, for taps conforming to a predetermined range of tap-definingtime intervals; identify tap at location x, open a tapping gesture; findand output touched location adjacent to location x; and monitor for endof tapping gesture by monitoring time intervals between taps andcomparing to predetermined range of tap-defining time intervals; whenends, close tapping gesture opened previously. One method for detectinga tapping gesture may include some or all of the steps illustrated inFIG. 6, suitably ordered e.g. as shown.

According to certain embodiments, the button (or any other graphicaluser interface element) might change its shape or color when theinteraction element/s e.g. finger/s hovers above that button (i.e.touches the screen at the button's location).

According to certain embodiments, exiting the hovering mode is triggeredby pressing a special button on a CE device, by lifting all fingers fromthe touchscreen for more than (e.g.) two seconds, or by performing aunique touch gesture such as but not limited to those described below.

Pure software solutions exist which perform the task of displaying avisual cue indicative of user's interactions with a handheld touchscreen, at the application level. These solutions render the displayedcontent themselves and then overlay the visual cue on top. A particulardisadvantage of these solutions is that they are restricted to thosefile types that can be rendered by the application. For example, animage requires a different rendering mechanism than a pdf document does;the application may not handle both. These applications are usuallyslower than the native viewer (which is usually hardware accelerated)for the same file types. In contrast, according to certain embodimentsof the current invention, the visual cue image is not handled at theapplication level and instead is appended to a lower software layercomprising already rendered surfaces of the displayed file. Thereby, thevisual cue becomes independent of the displayed interface including thefile type thereof, and rendering is entrusted to the nativeapplications.

A particular advantage of certain embodiments is the following: when aCE device is used by itself without using an external display, duringusage of a touch interface as an input device, the location of thefinger used for input is within the user's natural field of vision. Butwhen the user interacts with the CE device using a touch interface whilelooking at a remote display, this is no longer true and the user has nofeedback where his or her fingers are on the touch interface, forcingthe user to shift his or her gaze from the CE device to the remotedisplay and back many times.

It is appreciated that typically, as shown in FIGS. 1 a-1 c, videocontent shown on a larger, external or even remote screen such as a TVscreen, is identical to that displayed by, and is provided by, the handheld touch screen device (e.g. CE device) via a suitable connectionwhich may be wired or wireless. Any suitable technology, including butnot limited to picoprojector or Wifi technology, may be employed toimplement this arrangement.

Also provided is a computer program comprising computer program codemeans for performing any of the methods shown and described herein whensaid program is run on a computer; and a computer program product,comprising a typically non-transitory computer-usable or -readablemedium or computer readable storage medium, typically tangible, having acomputer readable program code embodied therein, said computer readableprogram code adapted to be executed to implement any or all of themethods shown and described herein. It is appreciated that any or all ofthe computational steps shown and described herein may becomputer-implemented. The operations in accordance with the teachingsherein may be performed by a computer specially constructed for thedesired purposes or by a general purpose computer specially configuredfor the desired purpose by a computer program stored in a typicallynon-transitory computer readable storage medium.

Any suitable processor, display and input means may be used to process,display e.g. on a computer screen or other computer output device,store, and accept information such as information used by or generatedby any of the methods and apparatus shown and described herein; theabove processor, display and input means including computer programs, inaccordance with some or all of the embodiments of the present invention.Any or all functionalities of the invention shown and described herein,such as but not limited to steps of flowcharts, may be performed by aconventional personal computer processor, workstation or otherprogrammable device or computer or electronic computing device orprocessor, either general-purpose or specifically constructed, used forprocessing; a computer display screen and/or printer and/or speaker fordisplaying; machine-readable memory such as optical disks, CDROMs,magnetic-optical discs or other discs; RAMs, ROMs, EPROMs, EEPROMs,magnetic or optical or other cards, for storing, and keyboard or mousefor accepting. The term “process” as used above is intended to includeany type of computation or manipulation or transformation of datarepresented as physical, e.g. electronic, phenomena which may occur orreside e.g. within registers and/or memories of a computer or processor.The term processor includes a single processing unit or a plurality ofdistributed or remote such units.

The above devices may communicate via any conventional wired or wirelessdigital communication means, e.g. via a wired or cellular telephonenetwork or a computer network such as the Internet.

The apparatus of the present invention may include, according to certainembodiments of the invention, machine readable memory containing orotherwise storing a program of instructions which, when executed by themachine, implements some or all of the apparatus, methods, features andfunctionalities of the invention shown and described herein.Alternatively or in addition, the apparatus of the present invention mayinclude, according to certain embodiments of the invention, a program asabove which may be written in any conventional programming language, andoptionally a machine for executing the program such as but not limitedto a general purpose computer which may optionally be configured oractivated in accordance with the teachings of the present invention. Anyof the teachings incorporated herein may wherever suitable operate onsignals representative of physical objects or substances.

The embodiments referred to above, and other embodiments, are describedin detail in the next section.

Any trademark occurring in the text or drawings is the property of itsowner and occurs herein merely to explain or illustrate one example ofhow an embodiment of the invention may be implemented.

Unless specifically stated otherwise, as apparent from the followingdiscussions, it is appreciated that throughout the specificationdiscussions, utilizing terms such as, “processing”, “computing”,“estimating”, “selecting”, “ranking”, “grading”, “calculating”,“determining”, “generating”, “reassessing”, “classifying”, “generating”,“producing”, “stereo-matching”, “registering”, “detecting”,“associating”, “superimposing”, “obtaining” or the like, refer to theaction and/or processes of a computer or computing system, or processoror similar electronic computing device, that manipulate and/or transformdata represented as physical, such as electronic, quantities within thecomputing system's registers and/or memories, into other data similarlyrepresented as physical quantities within the computing system'smemories, registers or other such information storage, transmission ordisplay devices. The term “computer” should be broadly construed tocover any kind of electronic device with data processing capabilities,including, by way of non-limiting example, personal computers, servers,computing system, communication devices, processors (e.g. digital signalprocessor (DSP), microcontrollers, field programmable gate array (FPGA),application specific integrated circuit (ASIC), etc.) and otherelectronic computing devices.

The present invention may be described, merely for clarity, in terms ofterminology specific to particular programming languages, operatingsystems, browsers, system versions, individual products, and the like.It will be appreciated that this terminology is intended to conveygeneral principles of operation clearly and briefly, by way of example,and is not intended to limit the scope of the invention to anyparticular programming language, operating system, browser, systemversion, or individual product.

Elements separately listed herein need not be distinct components andalternatively may be the same structure.

Any suitable input device, such as but not limited to a sensor, may beused to generate or otherwise provide information received by theapparatus and methods shown and described herein. Any suitable outputdevice or display may be used to display or output information generatedby the apparatus and methods shown and described herein. Any suitableprocessor may be employed to compute or generate information asdescribed herein e.g. by providing one or more modules in the processorto perform functionalities described herein. Any suitable computerizeddata storage e.g. computer memory may be used to store informationreceived by or generated by the systems shown and described herein.Functionalities shown and described herein may be divided between aserver computer and a plurality of client computers. These or any othercomputerized components shown and described herein may communicatebetween themselves via a suitable computer network.

In accordance with an aspect of the presently disclosed subject matter,there is provided a method for interacting with a smart phoneapplication, the method being operative in conjunction with an externaldigital display having a gazed-at display screen, on which a first imageis provided which is based on another image produced by a touchscreenbased hand held device, the method including: in a first, normal,default mode of operation, sensing a user's interactions with thetouchscreen based hand held device and modifying a display on thetouchscreen based hand held device accordingly; in a second, hovering,mode of operation, triggered by a trigger event and terminated by arelease event; sensing the user's interactions with the touchscreenbased hand held device and generating a representative display of theuser's interactions superimposed on the first image; and detecting atleast one predetermined compound touch event generated by the user'sinteraction with the touchscreen, including a first touch eventcomponent indicative of a touchscreen location and a second touch eventcomponent indicative of an operative action to be performed within thenormal mode of operation, and performing the operative action indicatedby the second touch event component at the touchscreen locationindicated by the first touch event component.

In accordance with an embodiment of the presently disclosed subjectmatter, there is further provided a method wherein said generating arepresentative display includes providing an indication, on a displaydevice controlling the gazed-at screen, of at least one finger locationon the touchscreen based hand held device thereby allowing a user togenerate touch screen events which control the display device's displayoutput.

In accordance with an embodiment of the presently disclosed subjectmatter, there is further provided a method wherein the providing anindication occurs only responsive to a trigger event initiated by theuser of the touch screen and wherein the trigger event comprises atleast one touch screen event representing a touch gesture in which theuser touches the touch screen.

In accordance with an embodiment of the presently disclosed subjectmatter, there is further provided a method, where the touch gesturecomprises placing two fingers on the touch screen at any location,followed by a drag of only one finger.

In accordance with an embodiment of the presently disclosed subjectmatter, there is further provided a method, where the touch gesturecomprises swiping two fingers in vertical movement in oppositedirections.

In accordance with an embodiment of the presently disclosed subjectmatter, there is further provided a method, where the touch gesturecomprises dragging a single finger from a predetermined location on atouchscreen.

In accordance with an embodiment of the presently disclosed subjectmatter, there is further provided a method, where the release eventcomprises a press of one of the CE device physical buttons.

In accordance with an embodiment of the presently disclosed subjectmatter, there is further provided a method, where the release eventcomprises a touch gesture in which the user touches the touch screen.

In accordance with an embodiment of the presently disclosed subjectmatter, there is further provided a method, where the touch gesturecomprises lifting all fingers from the touch screen.

In accordance with an embodiment of the presently disclosed subjectmatter, there is further provided a method, where the touch gesturecomprises placing an additional finger on the touch screen in anylocation.

In accordance with an embodiment of the presently disclosed subjectmatter, there is further provided a method, wherein generating arepresentative display comprises displaying a visual cursor overlay uponthe normal display of the CE device in question which is visible on theexternal gazed at surface.

In accordance with an embodiment of the presently disclosed subjectmatter, there is further provided a method, where at least onecharacteristic of the visual cursor provides a characterization of thetouch surface of at least one finger with the touch screen.

In accordance with an embodiment of the presently disclosed subjectmatter, there is further provided a method, where the visual cursor isdrawn by the apparatus used to render the remote display of the handheld device.

In accordance with an embodiment of the presently disclosed subjectmatter, there is further provided a method, where the visual cursor isdrawn using a display device which is additional to the display devicerendering the main remote display of the hand held device.

In accordance with an embodiment of the presently disclosed subjectmatter, there is further provided a method, wherein generating arepresentative display comprises generating an audio sound, sensible tothe user, of the at least one characteristic of which indicates to theuser the relative location of his finger on the touch screen.

In accordance with an embodiment of the presently disclosed subjectmatter, there is further provided a method, wherein generating arepresentative display comprises generating a haptic indication,sensible to the user, of the location of at least one finger on thetouch screen.

In accordance with an embodiment of the presently disclosed subjectmatter, there is further provided a method wherein the location of atleast one finger comprises locations of each of a plurality of fingersupon the touch screen.

In accordance with an embodiment of the presently disclosed subjectmatter, there is further provided a method wherein the characteristic ofthe visual cursor includes the shape thereof.

In accordance with an embodiment of the presently disclosed subjectmatter, there is further provided a method wherein the characteristic ofthe visual cursor includes the color thereof.

In accordance with an embodiment of the presently disclosed subjectmatter, there is further provided a method wherein the characterizationof the touch surface includes the size thereof.

In accordance with an embodiment of the presently disclosed subjectmatter, there is further provided a method wherein the characterizationof the touch surface includes the angular orientation thereof.

In accordance with an embodiment of the presently disclosed subjectmatter, there is further provided a method, where the trigger eventcomprises a key press.

In accordance with an embodiment of the presently disclosed subjectmatter, there is further provided a method where the CE device comprisesa smart phone.

In accordance with an embodiment of the presently disclosed subjectmatter, there is further provided a method where the CE device comprisesa media player.

In accordance with an embodiment of the presently disclosed subjectmatter, there is further provided a method wherein the characteristicincludes at least one of: pitch, height and volume.

In accordance with an embodiment of the presently disclosed subjectmatter, there is further provided a method wherein the generating arepresentative display includes providing an indication, on a displaydevice controlling the gazed-at screen, of at least one finger locationon the touchscreen based hand held device thereby allowing a user togenerate touch screen events which control the display device's displayoutput.

In accordance with an embodiment of the presently disclosed subjectmatter, there is further provided a method wherein the providing anindication occurs only responsive to a trigger event initiated by theuser of the touch screen.

In accordance with an embodiment of the presently disclosed subjectmatter, there is further provided a method wherein the trigger eventcomprises at least one touch screen event representing a touch gesturein which the user touches the touch screen.

In accordance with an embodiment of the presently disclosed subjectmatter, there is further provided a method comprising sniffing for thetrigger event while passing all touch screen events, other than thetrigger event, for touch screen event processing toward modification ofthe display output.

In accordance with an embodiment of the presently disclosed subjectmatter, there is further provided a method including, responsive to thetrigger event, grabbing all touch events generated by the user on thetouch screen during a time period, and for each of the touch events,providing a sensory cue for location of at least one finger upon thetouch screen.

In accordance with an embodiment of the presently disclosed subjectmatter, there is further provided a method including, responsive to thetrigger event, blocking all touch events generated by the user on thetouch screen during a time period, for touch screen event processingtoward modification of the display output.

In accordance with an embodiment of the presently disclosed subjectmatter, there is further provided a method wherein the time periodterminates upon occurrence of a release event.

In accordance with an embodiment of the presently disclosed subjectmatter, there is further provided a method wherein the release eventcomprises at least one touch screen event representing a touch gesturein which the user touches the touch screen.

In accordance with an embodiment of the presently disclosed subjectmatter, there is further provided a method wherein the operative actioncomprises a selected one of a range of predetermined operative actions.

In accordance with an embodiment of the presently disclosed subjectmatter, there is further provided a method wherein the sensing includesdetecting a gesture in which a plurality of fingers contact thetouchscreen and, within a predetermined time interval, some of theplurality of fingers are removed from the touchscreen whereas others ofthe plurality of fingers remain in contact with the touchscreen.

In accordance with an embodiment of the presently disclosed subjectmatter, there is further provided a method wherein the representativedisplay comprises a visual cue indicative of location of the user'sinteraction and wherein an initial position of the visual cue on thefirst image is exactly the initial position of the user's interaction onthe touch screen, suitably scaled to take into account difference insize between the touch screen and the gazed-at screen.

In accordance with an embodiment of the presently disclosed subjectmatter, there is further provided a method wherein the representativedisplay comprises a visual cue indicative of location of the user'sinteraction and wherein an initial position of the visual cue on thefirst image is predetermined irrespective of the initial position of theuser's interaction on the touch screen.

In accordance with an embodiment of the presently disclosed subjectmatter, there is further provided a method wherein the visual cue movesin each dimension to exactly reproduce the user's motions over thetouchscreen, scaled to take into account difference in size between thetwo screens.

In accordance with an embodiment of the presently disclosed subjectmatter, there is further provided a method wherein the visual cue movesin each dimension to exactly reproduce the user's motions over thetouchscreen, scaled by predetermined visual cue speed factorsirrespective of difference in size between the two screens.

In accordance with an aspect of the presently disclosed subject matter,there is yet further provided a system for adding a mode of operation toany of a population of applications for at least one touchscreen basedhand held device each having a normal mode of operation which includessensing a user's interactions with the touchscreen based hand helddevice and modifying a display on the touchscreen based hand held deviceaccordingly, the system comprising a software development functionalityoperative to add a hovering mode of operation to any of a population ofapplications for at least one touchscreen based hand held device,wherein the hovering mode of operation includes sensing the user'sinteractions with the touchscreen based hand held device and generatinga representative display of the user's interactions superimposed on afirst image displayed on an external display device.

In accordance with an embodiment of the presently disclosed subjectmatter, there is yet further provided a system wherein the softwaredevelopment functionality is part of an SDK (Software development kit).

In accordance with an embodiment of the presently disclosed subjectmatter, there is yet further provided a system wherein the softwaredevelopment functionality includes a library of trigger events, and atrigger event selection functionality operative to enable an applicationdeveloper to select an individual trigger event from thelibrary and toconfigure the application to pass from its normal mode of operation tothe hovering mode of operation responsive to the individual triggerevent selected.

In accordance with an embodiment of the presently disclosed subjectmatter, there is yet further provided a system wherein the softwaredevelopment functionality includes a library of release events, and atrigger event selection functionality operative to enable an applicationdeveloper to select an individual release event from the library and toconfigure the application to revert from the hovering mode of operationto its normal mode of operation responsive to the individual releaseevent selected.

In accordance with an embodiment of the presently disclosed subjectmatter, there is yet further provided a system wherein at least oneevent in the library of trigger events comprises a gesture.

In accordance with an embodiment of the presently disclosed subjectmatter, there is yet further provided a system wherein at least oneevent in the library of release events comprises a gesture.

In accordance with an embodiment of the presently disclosed subjectmatter, there is yet further provided a system wherein the softwaredevelopment functionality includes a library of cues indicating alocation on a screen display; and a cue selection functionalityoperative to enable an application developer to select an individual cuefrom the library and to configure the application to generate,responsive to sensed user's interactions with the touchscreen based handheld device, a representative display of the user's interactions,including the individual cue.

In accordance with an embodiment of the presently disclosed subjectmatter, there is yet further provided a system wherein the individualcue includes a visual cue.

In accordance with an embodiment of the presently disclosed subjectmatter, there is yet further provided a system wherein the visual cue issuperimposed on a first image displayed on an external display device,at a location representative of a location of the sensed user'sinteractions relative to the touchscreen based hand held device.

In accordance with an embodiment of the presently disclosed subjectmatter, there is yet further provided a system wherein the cue selectionfunctionality is also operative to enable an application developer toselect one of: a relative mode of movement of the visual cue; and anabsolute mode of movement of the visual cue.

In accordance with an embodiment of the presently disclosed subjectmatter, there is yet further provided a method wherein the hovering modeof operation is implemented in an Android system.

In accordance with an embodiment of the presently disclosed subjectmatter, there is yet further provided a method wherein the second,hovering, mode of operation is implemented on a Wi-Fi display.

In accordance with an embodiment of the presently disclosed subjectmatter, there is yet further provided a method wherein an underlyingview receives an onHoverEvent if an appropriate Listener is registeredwith the Android system.

In accordance with an embodiment of the presently disclosed subjectmatter, there is yet further provided a method wherein the first imageis displayed remotely.

In accordance with an embodiment of the presently disclosed subjectmatter, there is yet further provided a method wherein the gazed-atscreen comprises a television screen.

In accordance with an embodiment of the presently disclosed subjectmatter, there is yet further provided a method wherein softwarefunctionality residing in the hand-held device implements the second,hovering, mode of operation.

In accordance with an embodiment of the presently disclosed subjectmatter, there is yet further provided a system wherein the hovering modeof operation gives the user a cue to the place at which his fingers are,without performing any interaction with the interface of the handhelddevice.

In accordance with an aspectof the presently disclosed subject matter,there is yet further provided a method for interacting with an image,the method being operative in conjunction with a gazed-at screen, onwhich a first image is provided which is based on another image producedby a touchscreen based hand held device, the method including: in afirst, normal, mode of operation, sensing a user's interactions with thetouchscreen based hand held device and modifying a display on thetouchscreen based hand held device accordingly; in a second, hovering,mode of operation, sensing the user's interactions with the touchscreenbased hand held device and generating a representative display of theuser's interactions superimposed on the first image; and upon detectinga predetermined compound touch event generated by the user's interactionwith the touchscreen, passing from one of the modes of operation toanother, the compound touch event including a first event componentindicating a location on the touchscreen and a predetermined secondevent component which a user is capable of generating concurrently withthe first event component, the predetermined second event being reservedto trigger passage between the modes of operation.

In accordance with an embodiment of the presently disclosed subjectmatter, there is yet further provided a method wherein an interfacedisplayed on the hand held device is unaware of the second hovering modeof operation and interacts with the second hovering mode of operation asthough it were the first mode of operation.

In accordance with an embodiment of the presently disclosed subjectmatter, there is yet further provided a method and also comprisingsniffing for the trigger event while passing all touch screen events,other than the trigger event, for touch screen event processing towardmodification of the display output.

In accordance with an embodiment of the presently disclosed subjectmatter, there is yet further provided a method and also including,responsive to the trigger event, grabbing all touch events generated bythe user on the touch screen during a time period, and for each of thetouch events, providing a sensory cue for location of at least onefinger upon the touch screen.

In accordance with an embodiment of the presently disclosed subjectmatter, there is yet further provided a method including, responsive tothe trigger event, blocking all touch events generated by the user onthe touch screen during a time period, for touch screen event processingtoward modification of the display output.

In accordance with an embodiment of the presently disclosed subjectmatter, there is yet further provided a method wherein the time periodterminates upon occurrence of a release event.

In accordance with an embodiment of the presently disclosed subjectmatter, there is yet further provided a method wherein the release eventcomprises at least one touch screen event representing a touch gesturein which the user touches the touch screen.

In accordance with an aspect of the presently disclosed subject matter,there is yet further provided a system for interacting with a smartphone application, the system being operative in conjunction with anexternal digital display having a gazed-at display screen, on which afirst image is provided which is based on another image produced by atouchscreen based hand held device, the system interacting with a nativeinterface operative, in a first, normal, default mode of operation ofthe hand held device, for sensing a user's interactions with thetouchscreen based hand held device and modifying a display on thetouchscreen based hand held device accordingly, the system comprising: avirtual hovering hovering operative, in a second, hovering, mode ofoperation of the hand held device, triggered by a trigger event andterminated by a release event, for sensing the user's interactions withthe touchscreen based hand held device and generating a representativedisplay of the user's interactions superimposed on the first image; andfor detecting at least one predetermined compound touch event generatedby the user's interaction with the touchscreen, including a first touchevent component indicative of a touchscreen location and a second touchevent component indicative of an operative action to be performed withinthe normal mode of operation, and performing the operative actionindicated by the second touch event component at the touchscreenlocation indicated by the first touch event component.

In accordance with an aspect of the presently disclosed subject matter,there is yet further provided a computer program product, comprising acomputer usable medium having a computer readable program code embodiedtherein, the computer readable program code adapted to be executed toimplement a method for interacting with a smart phone application, themethod being operative in conjunction with an external digital displayhaving a gazed-at display screen, on which a first image is providedwhich is based on another image produced by a touchscreen based handheld device, the method including: in a first, normal, default mode ofoperation, sensing a user's interactions with the touchscreen based handheld device and modifying a display on the touchscreen based hand helddevice accordingly; in a second, hovering, mode of operation, triggeredby a trigger event and terminated by a release event: sensing the user'sinteractions with the touchscreen based hand held device and generatinga representative display of the user's interactions superimposed on thefirst image; and detecting at least one predetermined compound touchevent generated by the user's interaction with the touchscreen,including a first touch event component indicative of a touchscreenlocation and a second touch event component indicative of an operativeaction to be performed within the normal mode of operation, andperforming the operative action indicated by the second touch eventcomponent at the touchscreen location indicated by the first touch eventcomponent.

BRIEF DESCRIPTION OF THE DRAWINGS

Certain embodiments of the present invention are illustrated in thefollowing drawings:

FIGS. 1 a-1 c are simplified pictorial illustrations of apparatus usefulin conjunction with the methods of the present invention.

FIG. 2 is a simplified flowchart illustration of a method for providinga sensory cue representing an interaction with a touch screen basedinterface.

FIG. 3 a is a simplified pictorial illustration of a gesture applied toa touch screen which includes putting two fingers on the touch screenand swiping both fingers, e.g. vertically, typically in oppositedirections to one another, the gesture being particularly but notexclusively suited to use as an engagement event.

FIG. 3 b is a simplified pictorial illustration of a gesture applied toa touch screen which includes a press on the touch screen and atypically simultaneous tap with another finger, the gesture beingparticularly but not exclusively suited to use as an engagement event;in FIG. 3 b inter alia, a black (or hatched) circle denotes a locationon which a long press is applied by the user's finger/s while a whitecircle denotes a location on which a short tap is applied by the user'sfinger/s.

FIG. 3 c is a simplified pictorial illustration of a gesture,particularly but not exclusively suited to use as a trigger event,applied to a touch screen which includes putting two fingers on thetouch screen, and then lifting one finger, while continuing to apply theother finger to the screen.

FIG. 4 a is a simplified flowchart illustration of a dual mode method ofoperation for providing a sensory cue representing an interaction with aTouch Screen Based Interface and for normal operation with the touchscreen, according to a first embodiment of the present invention.

FIG. 4 b is a simplified flowchart illustration of a dual mode method ofoperation for providing a sensory cue representing an interaction with atouch screen based interface and for normal operation with the touchscreen, according to a second embodiment of the present invention.

FIG. 5 is a simplified flowchart illustration of a method for detectinga “rice-picking” gesture.

FIG. 6 is a simplified flowchart illustration of a method for detectinga tapping gesture.

Computational components described and illustrated herein can beimplemented in various forms, for example, as hardware circuits such asbut not limited to custom VLSI circuits or gate arrays or programmablehardware devices such as but not limited to FPGAs, or as softwareprogram code stored on at least one intangible computer readable mediumand executable by at least one processor, or any suitable combinationthereof. A specific functional component may be formed by one particularsequence of software code, or by a plurality of such, which collectivelyact or behave or act as described herein with reference to thefunctional component in question. For example, the component may bedistributed over several code sequences such as but not limited toobjects, procedures, functions, routines and programs and may originatefrom several computer files which typically operate synergistically.

Data can be stored on one or more intangible computer readable mediastored at one or more different locations, different network nodes ordifferent storage devices at a single node or location.

It is appreciated that any computer data storage technology, includingany type of storage or memory and any type of computer components andrecording media that retain digital data used for computing for aninterval of time, and any time of information retention technology, maybe used to store the various data provided and employed herein. Suitablecomputer data storage or information retention apparatus may includeapparatus which is primary, secondary, tertiary or off-line; which is ofany type or level or amount or category of volatility, differentiation,mutability, accessibility, addressability, capacity, performance andenergy use; and which is based on any suitable technologies such assemiconductor, magnetic, optical, paper and others.

DETAILED DESCRIPTION OF CERTAIN EMBODIMENTS

Reference is now made to FIGS. 1 a-1 c which illustrate apparatus usefulin conjunction with the methods of the present invention and togenerally self-explanatory FIG. 2 which illustrates a method forproviding a sensory cue for the position of digits in a touch screenbased interface provided in accordance with certain embodiments of thepresent invention. Generally, the user interface of the currentgeneration of hand held consumer electronic devices 5 (henceforth “handheld devices”), such as smart phones and media players, is touch driven.An emerging trend in this area is the beaming or remoting of the userinterface display of these devices onto external displays 10, such as bythe use of pico projectors 70, use of HDMI connectors 20, or use ofvideo streaming technologies to stream the UI of the CE device 5 into amedia streamer connected to a TV 20.

An emerging problem for designers of new CE devices, resulting fromthese two trends, is that while the physical screen of the CE device isused as a touch input device, the location of the finger or fingers usedfor touch input is naturally visible to the person interacting with theCE device, while when the user interacts with the CE device using atouch interface while looking at a remote display, this is no longertrue, and the user has no feedback where his or her fingers are on thetouch interface, forcing the user to shift his or her gaze from the CEdevice to the remote display and back many times. This is intrusive tothe user's concentration and is therefore undesirable.

In a typical implementation, a hand held device 5, such as a smart phoneor personal media device, is equipped with an embedded display (ascreen) which supports touch events as an input mechanism.

In a typical implementation, the same hand held device also supportssome means of displaying its user interface or a remote display. Thisremote display may show a replica of the embedded display or a differentUI or content.

In a favorable implementation, the means of the hand held device fordisplaying its display remotely allows the user to hold the hand helddevice at a distance from the actual display 10. This can be done, forexample, via the use of pico projectors that are internally part of somehand held devices today, or by use of technology to transmit viawireless a video stream of the display to remote TV or video streamer asa compressed video stream (encoded MPEG4 stream over UDP over Wifi, forexample).

In other implementations, the means of the hand held device 5 to providethe remote display 10 involves a physical connection, such as an HDMIconnector from the phone attached to the display. In addition,combinations and variations of the above are also possible, such as aphysical HDMI connection to a third party device that transmits thevideo stream produced by the hand held device over a wirelessconnection.

In a typical implementation, the user of the hand held device willinteract with the device UI using the touch interface, by putting one ormore fingers on the touch screen and performing various movements, orgestures, to indicate to the device the operations required by thedevice user.

In a typical implementation, the hand held device will be set up todisplay its user interface remotely as described above. The user willthen put his or her eyes on the remote display and perform the triggerevent to engage the system. The user will then interact with the systemby moving his/her finger across the touch screen and receiving a sensorycue notifying him where his/her finger is located. The interaction willend when the user performs the release event.

In a typical implementation, the system will be implemented asadditional computer software code that will be added to the code thathandles the reading and handling of touch screen events from thehardware and dispatching it to higher levels of code in the system.

Such code typically reads touch events created by the touch screenhardware and, based on some logic, notifies higher layers of code that atouch, a swipe or other gesture have been received. In this typicalimplementation, this gesture detection and event relay logic will beexpanded to include the detection of the trigger event, as well as theadditional behavior outlined as follows.

In some implementations this trigger even might be pressing of a keydesignated for this function in a specific context on the hand helddevice.

In another implementation, the user of the hand held device will performa touch gesture to engage the system. This implementation is very easyfor the user, but requires an additional button on the hand held device.

In some implementations, the gesture is comprised of putting of twofingers on the touch screen, and then lifting one finger, whilecontinuing to press the other finger onto the touch screen. Thisimplementation is relatively easy for the user, but it is possible thatthe trigger event will make it difficult to implement other common touchUI gestures, such as pinch and zoom.

In other implementations, the gesture is comprised of simply putting asingle finger on the touch screen. This implementation is very easy forthe user, but requires the user to adjust to a different meaning oftouch gesture when using a remote display and otherwise, as commonlyputting a finger on a touch display is reserved to a different action(click, swipe or drag) in many existing touch user interfaceimplementations.

In one implementation, the gesture will be comprised of putting twofingers on the touch screen and swiping each one horizontally inopposite directions to each other. This implementation has the advantageof not requiring additional keys on the touch device, while at the sametime not requiring changing the meaning of common gestures commonly usedin touch based user interface implementations.

Once the trigger event has been registered by the system, the systemwill start providing a sensory cue for the location of the finger on thetouch screen to the hand held user. This will be done by the programcode creating a sensory stimulus for the hand held device user such asthe nature of the stimulus is tied to the location, and optionally otherparameters related to the finger or fingers of the user on the touchscreen.

In a typical implementation, the sensory stimulus is visual—the systemwill add a visual cursor on top of the display corresponding to thelocation of the finger or fingers on the touch screen display. In someimplementations the visual cursor might be translucent, thus allowing tosee the display underneath, or it might be opaque. The visual cursormight have any shape, such as, for example, a cross, a hand, an arrow ora point.

In a typical implementation of this sort, the visual cursor will bedrawn using the same mechanism that is used to draw the remote display,that is the system will instruct the part of the software of the handheld device that is responsible to draw the screen, to overlay thevisual cursor on top of the display. In such an implementation, thecursor will be displayed by whatever mechanism that is drawing theremote display, for example a pico projector, video streaming or HDMIoutput, as the case may be.

In other implementations, the visual cursor may be overlaid on top ofthe remote display by a different mechanism than is used to draw theremote display itself An example of such an implementation would be theprojecting of a visual cursor using a secondary pico projector,synchronized to the main pico projector that is rendering the mainremote display of the hand held device.

In one implementation, the touch screen sensor may report additionalinformation, in addition to the location of the finger or fingers on thetouch screen. Such information can include, for example, the size of thefinger press and the angle of the finger. This additional informationcan be used to display a visual cursor whose exact shape and size changeto reflect the exact size and angle of the finger on the touch screen.

In another possible implementation, the sensory stimulus isauditory—once engaged, the system will generate a sound, whoseproperties, such as pitch, tempo or volume, change in response to thelocation of the finger on the touch screen.

As an example, one possible implementation may encode the distance ofthe finger from the top of the touch screen in the pitch of the sound,while the distance from the leftmost side of the screen is encoded inthe volume of the sound. In this implementation, pressing a finger onthe left top corner of the screen will produce a low volume high pitchedsound, while moving a finger along the touch screen diagonal will raisethe sound volume and lower its pitch, accordingly.

In yet another implementation, the sensory stimulus is haptic—onceengaged, the system uses a mechanical vibrator built into the hand helddevice to convey, via haptic feedback, the location of the finger on thetouch screen.

As an example, one possible implementation may encode the location ofthe finger on the touch screen, as a series of pairs of vibrations,separated by a pause. The length of the first vibration corresponds tothe distance of the finger from the top of the touch screen, whereas thelength of the second vibration that follows in the set corresponds tothe distance of the finger from the left-most side of the touch screen.

In such an implementation as described above, holding a finger on thetop left corner of the touch screen will result in a series of two longsuccessive vibrations followed by a pause, where sliding a fingertowards the right most side of the screen, while keeping its relativeposition to the top of the screen, will result in a series of vibrationsto gradually change into pairs of long and short vibrations, followed bya pause—the first long vibration indicating that the finger is at thetop of the screen, whereas the following vibration that gets shorter andshorter as one moves one's finger as described, indicates that thefinger position is approaching the right-most side of the touch screen.

Many other implementations can be described using combinations andvariations of the various sensory inputs described above.

When the user of the hand held device has decided to perform anoperation with the touch screen (such as click an icon, perform a dragor pinch operation), he needs to disengage the system. This is done bythe user providing the release event.

In one implementation, this release event may be the pressing of a keyon the hand held device, that is dedicated to this function in thiscontext.

In another implementation, the hand held user may perform a touch screengesture, such as the release event.

One possible implementation of this release event gesture, may be theuser lifting all fingers off the touch screen. Another possibleimplementation of this release event gesture is the user adding anadditional finger to the touch screen beyond the one the system has beentracking thus far.

In any implementation, the system that tracks the touch screen sensorsreported events, will treat the release event as a signal to exit thefinger tracking mode and will, from that point forward, pass the touchinformation to the normal processing of the hand held device software,except that it will continue to track the touch screen event stream forthe trigger event, and if so, will engage the system anew, as describedabove.

Embodiments include:

-   -   1. A method and system for providing an a sensory cue for the        location of fingers on a touch device atop a display device.        -   1.1 Look for trigger event, while reverting all touch screen            events to normal processing.        -   1.2. Grab all touch events coming from touchscreen.        -   1.3. For each touch event, provide a sensory cue for            location of the finger or fingers upon the touch screen.        -   1.4. Block the touch event from further processing by the            OS.        -   1.5. Repeat step 2 to 5 until end of release event.        -   1.6. Go back to step 1.    -   2. The method and system of embodiment 1, where the trigger        event is the press of a key.    -   3. The method and system of embodiment 1, where the trigger        event is a touch gesture.    -   4. The method and system of embodiment 3, where the trigger        event is the touch gesture of placing two fingers on the touch        screen at any location, followed by a drag of a single finger.    -   5. The method and system of embodiment 3, where the trigger        event is a touch gesture where two fingers are swiped in        vertical movement in opposite directions.    -   6. The method and system of embodiment 3, where the trigger        event is a touch gesture where a single finger is dragged.    -   7. The method and system of embodiment 1, where the release        event is a press of a key.    -   8. The method and system of embodiment 1, where the release        event is a touch gesture.    -   9. The method and system of embodiment 8, where the release        event is a touch gesture of lifting all fingers from the touch        screen.    -   10. The method and system of embodiment 8, where the release        event is a touch gesture of placing an additional finger on the        touch screen in any location.    -   11. The method and system of embodiment 1, where the sensory cue        is the displaying of a visual cursor overlay upon the normal        display of the CE device in question.    -   12. The method and system of embodiment 11, where the visual        cursor shape and/or color provides a visual indication of the        exact touch surface of the finger or fingers with the touch        screen, including the size and angle of the area being touched.    -   13. The method and system of embodiment 11, where the visual        cursor is drawn using the same method that is used to render the        remote display of the hand held device.    -   14. The method and system of embodiment 11, where the visual        cursor is drawn using a secondary display device to the display        device that is rendering the main remote display of the hand        held device.    -   15. The method and system of embodiment 1, where the sensory cue        is an audio sound, whose pitch, height or volume indicates to        the user the relative location of his/her finger on the touch        screen.    -   16. The method and system of embodiment 1, where the sensory cue        is a haptic indication of the location of the finger or fingers        on the touch screen.

Certain embodiments of the present invention are now described indetail.

In one typical embodiment, computer software added to the conventionalCE device code reads touch events created by the touch screen hardwareand, based on suitable gesture detection and event relay logic, notifieshigher layers of code that a touch, a swipe or other gesture has beenreceived. In this typical embodiment, this gesture detection and eventrelay logic may be expanded to include detection of a trigger event andof a release event as described herein.

The trigger event may be effected using one of the sensors embedded inthe CE device:

a. In one embodiment, this trigger event might be pressing of anadditional key designated for this function in a specific context (suchas volume, or camera keys in the context of remotely displaying thescreen content on an external screen) on the hand held CE device.

b. In another embodiment the trigger event may comprise a touch gesturesuch as the following touch gesture: putting two fingers on the touchscreen, and then lifting one finger, while continuing to press the otherfinger onto the touch screen. This trigger event is also termed herein“rice-picking”. An example of this gesture is illustrated in FIG. 3 c.An example method for detecting this gesture may include some or all ofthe steps of the method of FIG. 5, suitably ordered e.g. as shown.

c. In another embodiment the user might trigger other sensors present inthe CE device in order to enter the hovering mode, for example by doingactions like shaking the device, emitting some sound, or blocking thelight detector of the CE device, which might signal entry into thehovering mode.

d. In other embodiments, the gesture comprises putting a single fingeron the touch screen. This embodiment is very easy for the user, butrequires the user to keep in mind the different meaning of a touchgesture when using a remote display as opposed to otherwise e.g. asopposed to during normal interaction with the CE device, since in thelatter mode, placing a finger on a touch display is reserved for adifferent action (click, swipe or drag) in many existing touch userinterface embodiments.

e. In still another embodiment, the trigger event gesture may comprise a“swiping” gesture including, e.g. putting two fingers on the touchscreen and swiping each one horizontally in opposite directions to eachother (e.g. as illustrated in FIG. 3 a). This embodiment has theadvantage of not requiring additional keys on the CE device while at thesame time not requiring changing the meaning of common gestures commonlyused in touch base user interfaces. Any suitable conventional swipinggesture detection methodology may be employed to detect the “swiping”gesture as illustrated in FIG. 3 a.

In another embodiment the trigger even comprises a gesture of puttingone finger (or more) in a dedicated or reserved location on the screen(e.g one of the corners or a specific one of the 4 corners); placing thefinger in any other place will not trigger reverting to the hoveringmode. Typically, a corner or side location is selected to be reservedfor triggering, since the middle locations are very frequently used innative interfaces hence may cause an unacceptable level of ambiguityhence interfering with or detracting from the user experience. If alocation or gesture selected to be reserved for (say) triggering is aptto cause ambiguity because it is needed also for the native interfaces,the triggering event (say) may be defined as a combination of locationand gesture. So, for example, a triggering event might be defined as agesture as in FIG. 3 a, which is effected in a location 5 pixels or lessfrom any corner of the screen, or a release event might be defined as agesture as in FIG. 3 b, which is effected in a location 10 pixels orless from the left edge of the screen.

Once the trigger event has been received by the CE's operating system, asensory cue is provided to represent the location of the interactionelement/s e.g. finger/s on the native interface (e.g. spatial domainrepresented by the CE device's touch screen). This may be done by theprogram code creating a sensory stimulus for the hand held device userwhich represents the location of and optionally other attributes of, thefinger or fingers of the user on the touch screen.

In one embodiment, the sensory stimulus is visual e.g. a visual cursoris superimposed on top of the display, which corresponds to the locationof the finger or fingers on the touch screen display. In someembodiments the visual cursor might be translucent, thus allowing thedisplay underneath to be visible, or it might be opaque. The visualcursor might have any shape, (such as but not limited to a cross, ahand, an arrow, a point, or a circle).

The visual cursor may be drawn using the same technology that is used todraw the remote display, e.g. the system may instruct the module of thesoftware of the hand held CE device that is responsible to draw thescreen, to overlay the visual cursor on top of the display. In such anembodiment the cursor may be displayed by whatever technology is drawingthe remote display, such as but not limited to a pico projector, videostreaming or HDMI output. In one embodiment the visual cue will bevisible on the primary display of the CE device, while in anotherembodiment it will be visible only on the external display.

According to one embodiment, the CE device's touch screen's inherentsensor may report additional information, in addition to the location ofthe finger or fingers on the touch screen. Such information can include,for example, the size of the finger press and/or the angle of the fingere.g. relative to the axes of the CE device's touch screen. Thisadditional information can be used to display a visual cursor whosecharacteristics e.g. shape and/or size and/or color change to reflectthe pressure and/or angle of the finger on the touch screen.

In another possible embodiment, the sensory stimulus may be auditory.Once in hovering mode, the CE device may generate a sound, whoseproperties such as pitch, tempo or volume change in response to thelocation of the finger on the touch screen.

As an example, one possible embodiment may encode the distance of thefinger from the top of the touch screen in the pitch of the sound, whilethe distance from the left-most side of the screen is encoded in thevolume of the sound. For example, pressing a finger on the left topcorner of the screen may produce a low volume high pitched sound,pressing a finger on the right bottom corner of the screen may produce ahigh volume low pitched sound, and moving the finger along the touchscreen diagonal may raise the sound volume and lower its pitch, or viceversa, accordingly.

In yet another embodiment, the sensory stimulus is haptic—once engaged,the system uses a mechanical vibrator built into the hand held CE deviceto convey, via haptic feedback, the location of the finger on the touchscreen.

As an example, one possible embodiment may encode the location of thefinger on the touch screen, as a sequence of pairs of vibrations,separated by a pause. The length of the first vibration corresponds tothe distance of the finger from the top of the touch screen, whereas thelength of the second vibration that follows in the set corresponds tothe distance of the finger from the left most side of the touch screen.

For example, holding the finger on the top left (say) corner of thetouch screen may result in a sequence of two long successive vibrationsfollowed by pause, where sliding the finger towards the right most sideof the screen while keeping its position relative to the top of thescreen, may result in a sequence of vibrations including pairs ofrelatively long and short vibrations respectively, each followed by apause—the first long vibration indicating that the finger is at the topof the screen, and remains generally uniform. The second vibration getsshorter and shorter as one moves one's finger as described, indicatingthat the finger position is approaching the right most side of the touchscreen.

Other embodiments may use any suitable combination of the varioussensory inputs described above.

FIGS. 4 a and 4 b are simplified flowchart illustrations of dual modemethods of operation for providing a sensory cue representing aninteraction with a touch screen based interface and for normal operationwith the touch screen, according to first and second embodiments of thepresent invention respectively. According to a preferred embodiment ofthe present invention, all functionality provided in accordance withembodiments of the present invention is performed by and resides insoftware on the hand held device. However, alternatively, some or allfunctionality may reside in remote locations e.g. in the controlapparatus for the remote display.

The hovering mode is now described, i.e. how the user moves theinteracting elements according to certain embodiments:

In one, “absolute”, embodiment, the way of moving the element/s e.g.finger/s may be called absolute movement. When the sensory cue is firstprovided, it will be provided in the location where the finger touchedthe touchscreen when entering the hovering mode. For example if thefinger touched the upper left corner, then this will be the location atwhich the sensory cue will be given (for example, at which the redcircle (e.g.) will be drawn).

This may be true during the movement of the finger on the touchscreen—atevery moment the sensory cue will be displayed in the exact location ofthe finger, such that if the finger moves to the bottom right corner onthe touchscreen, the visual cue will accordingly move to the bottomright corner on the external display.

In another, “relative” embodiment, the way of moving the element/s e.g.finger/s may be called relative movement. In this scheme, the relationbetween the point touched on the screen and the location of the sensorycue is determined “contextually” i.e. relative to the stored location ofthe sensory cue, which was stored from the sensory cue's previousmovements. In this scheme, the relation between the point touched on thescreen and the location of the sensory cue typically relies on theprevious movements of the interaction element/s e.g. finger/s Forexample, moving the finger from the left border of the touchscreen toits right border may only move the sensory cue a quarter of the screen,such that two such left to right swipes are needed so that the sensorycue may move the whole screen. Typically the sensory cue location ispreserved between the two swipes for a short period of time.

After moving the interaction element to the desired place (for exampleabove the button that the user wants to click), typically the user ofthe hand held device may want to interact with the CE device nativeinterface (such as click a button or perform a drag or pinchoperation).The user may instruct the system that such interaction iswanted, by effecting an event termed herein an “engagement event” suchas but not limited to a mouse click.

In one embodiment there is no engagement event. Instead, the manner ofthe interaction with the CE device native interface will be slightlychanged, to generate uniqueness recognizable by the CE device'soperating system, in that placing a finger on the screen will cause avisual cue to appear at the finger-identified location, instead ofcausing interaction with the button at the finger's location on thescreen. For example:

a. clicking of a button on the touch screen, as part of the touchscreen,will be effected by placing an additional finger on the touch screenwhile holding still a finger which moves a virtual interaction apparatuson the screen. The virtual interaction apparatus may include any virtualdevice (e.g. mouse cursor, visual cue such as red dot, etc.) which auser may use to perform interaction with the interface. The virtualdevice typically has a visible representation which may be the same asthe visual cue described herein.

b. alternatively, a special physical button, or button combination (suchas volume or camera buttons on the CE), may be pressed. This may requirea user to adjust to a different mode of interacting with the underlyingnative interface.

Typically, a user may press such a physical button which the CE devicewhen in hovering mode interprets differently than when the CE device isin its normal mode. For example when in its normal mode, a CE device mayopens its camera application when the “camera” physical button ispressed. However when in hovering mode, the CE device may interpret thecamera button press by the user as an engagement event as describedherein and may effect a suitable interaction with a native interfacesuch as, say an Iphone desktop. So, when in hovering mode, the CE devicemay respond to a camera button press by opening a game application whoseicon lies below the visual cue's (e.g. red dot's) location when thecamera button is pressed. Or, if due to previous operation in normalmode, the game environment of, say, the Iphone has been entered, and ashooting game has been initiated, then the CE device, when in hoveringmode, may respond to a camera button press not by opening the cameraapplication but instead by “shooting” at the touch screen location whichlies below the visual cue's (e.g. red dot's) location when the camerabutton is pressed.

In another embodiment the engagement event comprises a timed windowwhich initiates when a release event occurs, for performing theinteraction with the CE device native interface in a conventional way.In the timed window, the system may remember, for a certain period oftime, the location of the interaction element/s e.g. finger/s, and whenthe user performs an interaction (such as a tap on a screen for clickinga button) within the timed window, the interaction may be applied to thestored location on the screen of the interaction element/s e.g. finger/srather than being applied to the location at which the user actuallyperformed the interaction. Entering the timed window is triggered byperforming a release event.

In one embodiment, this release event may be the pressing of some key onthe hand held CE device, that is dedicated to the release function, atleast in this context.

In another embodiment, a particular touch screen gesture may beinterpreted as the release event. The gesture can either be a uniquegesture not used in other applications, e.g. “rice-picking” as shown inFIG. 3 c herein, or be “reserved” in that the user knows he is notallowed to use them except to designate a release event.

One possible embodiment of this release event gesture, may be liftingall fingers off the touch screen. Another possible embodiment of thisrelease event gesture is the user adding an additional finger to thetouch screen beyond the one the system has been tracking so far.

In any embodiment, the inherent module within the CE device that tracksthe touch screen sensors' reported events, may treat the release eventas a signal to exit the hovering mode, and may, from that point forward,relay the touch information to the normal processing of the hand helddevice software, except that it may continue to monitor or track thetouch screen event stream for the trigger event, and upon encounteringsuch, will transmit the CE operating system into the hovering mode.

Typically, for a release event, touch screen location is not relevantwhereas as for an engagement event, touch screen location is relevantand is typically determined before the gesture constituting theengagement event occurs e.g. by suitable prior positioning of the visualcue. For a trigger event too, touch screen location is typicallyrelevant.

According to an embodiment of the present invention, a generic softwarepackage is provided that may be available through an SDK (Softwaredevelopment kit) such that an application developer will have optionssuch as but not limited to some or all of the following: choosingwhether the hovering mode will be enabled or not in his application,choose the way that the trigger event and the release events are done,for example assign a gesture to each event, and choose the look and feelof the visual cue and more.

Methods for analyzing the touchscreen movement events operative inaccordance with certain embodiments of the present invention areillustrated in FIGS. 4 a-4 b; it is appreciated that some or all of theillustrated steps, suitably ordered e.g. as shown, may be provided.

FIG. 5 is a simplified flowchart illustration of an example method forperforming the gesture detection step 425 of FIG. 4 a, for a“rice-picking” gesture e.g. as shown in FIG. 3 c. FIG. 6 is a simplifiedflowchart illustration of an example method for performing the gesturedetection step 425 of FIG. 4 a, for a tapping gesture e.g. as shown inFIG. 3 b.

Referring now to FIG. 4 b, it is appreciated that generally, a gestureserving as a trigger (or release) event may comprise more than one part,e.g. may itself comprise a sequence of more than one gesture, forexample, as shown in FIG. 3 a: A 1st gesture or first part, as indicatedby sketch I—touching the screen with two fingers; followed by a 2ndgesture or second part, as indicated by sketch II—swiping the fingers inopposite directions. When this is the case, the gesture is “completed”if both or all gestures in the sequence are performed by the user anddetected by the system, which then proceeds under the assumption that atrigger (or release) event has occurred. In a 2-gesture sequence, forexample, gesture detection includes detecting the start of the firstgesture in the sequence, tracking from the start of the first gestureuntil its termination, and then detecting the final part i.e. the secondgesture in the sequence, in the same manner. A time period may bepre-defined to indicate the maximum amount of time which may elapse fromwhen the first gesture finishes till when the second gesture starts.

It is appreciated that any of the gestures described herein may be used,e.g. as per a predetermined scheme, either as a trigger event triggeringa move from normal mode to hovering mode (as a result of which move,typically, a visual cue or other indicator appears e.g. on thedisplaying screen); or as a release event triggering a move fromhovering mode to normal mode (as a result of which move, typically, thevisual cue or other indicator disappears e.g. from the displayingscreen), or in fact as a “toggle” serving both as a trigger event and asa release event. Any event e.g. gesture described herein as a triggerevent may also serve, alternatively or in addition, as a release event,and vice versa. Typically, a trigger event is selected to be a gesturewhich terminates in the presence of only a single finger on the screen,e.g. the gesture of FIG. 3 c, since if this is the case, the position ofthe visual cue is clearly determined by the position of the sole fingerremaining on the screen as the gesture terminates. Alternatively, atrigger event is selected to be a gesture which terminates in thepresence of more than a single finger on the screen, e.g. the gesture ofFIG. 3 a. If this is the case, the position of the visual cue isdetermined by the position of a predetermined one of the more than onefinger(s) remaining on the screen as the gesture terminates, such as thefinger closer to the top and/or right of the screen.

Typically, gestures e.g. as described herein are utilized as triggerand/or release events, whereas engagement events may, for example,comprise clicking a virtual button on the touch screen, responsive towhich the interaction indicated by the virtual button is carried out, atthe location indicated by the visual (say) cue.

It is appreciated that any suitable scheme may be used to map a changeof position of the finger (e.g.) on the touch screen, to a change ofposition of the visual cue on the screen being used to display.

For example, according to a first, “absolute” embodiment of theinvention:

-   a. the initial position of the visual cue on the displaying screen    is exactly that of the finger on the touch screen, suitably scaled    to take into account difference in size between the two screens; and-   b. Each delta-x, delta-y move of the finger on the touch screen is    represented by a scaled delta-X, delta-Y move of the visual cue on    the displaying screen, where delta-X, delta-Y are derived from    delta-x, delta-y e.g. again simply by scaling to take into account    difference in size between the two screens.

However, according to a more general, “relative” embodiment of theinvention,

-   a. the initial position of the visual cue on the displaying screen    may be a predetermined position unrelated to the initial position of    the finger on the touch screen, e.g. the middle of the screen;    and/or-   b. Each delta-x, delta-y move of the finger on the touch screen is    represented by a scaled delta-X, delta-Y move of the visual cue on    the displaying screen, where the factors by which delta-X, delta-Y    differ respectively from delta-x, delta-y are not simple scaling    factors which take into account difference in x and y dimensions    between the two screens. Instead, each of these factors is a    pre-determined parameter; these pre-determined parameters may be    termed the x-velocity and y-velocity of the visual cue and they    determine whether the visual cue will “zoom across the displaying    screen” responsive to even a tiny motion of the finger (e.g.) on the    touch screen, or conversely whether the finger may even need to    traverse the entire width or length of the touch screen several    times in order to cause the visual cue to move from one side of the    displaying screen to the other.

Optionally, the image shown on the large screen may be formed asfollows:

a. use native PDF application which is provided today as part of thehandheld, to generate bits for display which represent the desired imageon the large screen other than the image of the visual cue; and

b. superimpose onto these bits, additional bits representing the visualcue e.g. red dot

A particular advantage of this option is that the above method is notadversely affected by the size of the PDF file required to represent theimage on the large screen.

It is appreciated that terminology such as “mandatory”, “required”,“need” and “must” refer to implementation choices made within thecontext of a particular implementation or application describedherewithin for clarity and are not intended to be limiting since in analternative implantation, the same elements might be defined as notmandatory and not required or might even be eliminated altogether.

It is appreciated that software components of the present inventionincluding programs and data may, if desired, be implemented in ROM (readonly memory) form including CD-ROMs, EPROMs and EEPROMs, or may bestored in any other suitable typically non-transitory computer-readablemedium such as but not limited to disks of various kinds, cards ofvarious kinds and RAMs. Components described herein as software may,alternatively, be implemented wholly or partly in hardware, if desired,using conventional techniques. Conversely, components described hereinas hardware may, alternatively, be implemented wholly or partly insoftware, if desired, using conventional techniques.

Included in the scope of the present invention, inter alia, areelectromagnetic signals carrying computer-readable instructions forperforming any or all of the steps of any of the methods shown anddescribed herein, in any suitable order; machine-readable instructionsfor performing any or all of the steps of any of the methods shown anddescribed herein, in any suitable order; program storage devicesreadable by machine, tangibly embodying a program of instructionsexecutable by the machine to perform any or all of the steps of any ofthe methods shown and described herein, in any suitable order; acomputer program product comprising a computer useable medium havingcomputer readable program code, such as executable code, having embodiedtherein, and/or including computer readable program code for performing,any or all of the steps of any of the methods shown and describedherein, in any suitable order; any technical effects brought about byany or all of the steps of any of the methods shown and describedherein, when performed in any suitable order; any suitable apparatus ordevice or combination of such, programmed to perform, alone or incombination, any or all of the steps of any of the methods shown anddescribed herein, in any suitable order; electronic devices eachincluding a processor and a cooperating input device and/or outputdevice and operative to perform in software any steps shown anddescribed herein; information storage devices or physical records, suchas disks or hard drives, causing a computer or other device to beconfigured so as to carry out any or all of the steps of any of themethods shown and described herein, in any suitable order; a programpre-stored e.g. in memory or on an information network such as theInternet, before or after being downloaded, which embodies any or all ofthe steps of any of the methods shown and described herein, in anysuitable order, and the method of uploading or downloading such, and asystem including server/s and/or client/s for using such; and hardwarewhich performs any or all of the steps of any of the methods shown anddescribed herein, in any suitable order, either alone or in conjunctionwith software. Any computer-readable or machine-readable media describedherein is intended to include non-transitory computer- ormachine-readable media.

Any computations or other forms of analysis described herein may beperformed by a suitable computerized method. Any step described hereinmay be computer-implemented. The invention shown and described hereinmay include (a) using a computerized method to identify a solution toany of the problems or for any of the objectives described herein, thesolution optionally include at least one of a decision, an action, aproduct, a service or any other information described herein thatimpacts, in a positive manner, a problem or objectives described herein;and (b) outputting the solution.

The scope of the present invention is not limited to structures andfunctions specifically described herein and is also intended to includedevices which have the capacity to yield a structure, or perform afunction, described herein, such that even though users of the devicemay not use the capacity, they are, if they so desire, able to modifythe device to obtain the structure or function.

Features of the present invention which are described in the context ofseparate embodiments may also be provided in combination in a singleembodiment.

For example, a system embodiment is intended to include a correspondingprocess embodiment. Also, each system embodiment is intended to includea server-centered “view” or client centered “view”, or “view” from anyother node of the system, of the entire functionality of the system,computer-readable medium, apparatus, including only thosefunctionalities performed at that server or client or node.

Conversely, features of the invention, including method steps, which aredescribed for brevity in the context of a single embodiment or in acertain order may be provided separately or in any suitablesubcombination or in a different order. “e.g.” is used herein in thesense of a specific example which is not intended to be limiting.Devices, apparatus or systems shown coupled in any of the drawings mayin fact be integrated into a single platform in certain embodiments ormay be coupled via any appropriate wired or wireless coupling such asbut not limited to optical fiber, Ethernet, Wireless LAN, HomePNA, powerline communication, cell phone, PDA, Blackberry GPRS, Satelliteincluding GPS, or other mobile delivery. It is appreciated that in thedescription and drawings shown and described herein, functionalitiesdescribed or illustrated as systems and sub-units thereof can also beprovided as methods and steps therewithin, and functionalities describedor illustrated as methods and steps therewithin can also be provided assystems and sub-units thereof. The scale used to illustrate variouselements in the drawings is merely exemplary and/or appropriate forclarity of presentation and is not intended to be limiting.

The invention claimed is:
 1. A method for interacting with a touch screen based hand held device, the method being operative in conjunction with an external digital display having a gazed-at display screen, on which a first image is provided which is based on another image produced by a touch screen based hand held device, the method including: in a first, normal, default mode of operation, sensing a user's interactions with the touch screen based hand held device and modifying a display on the touchscreen based hand held device accordingly; in a second, hovering, mode of operation, triggered by a trigger event and terminated by a release event: sensing the user's interactions with the touch screen based hand held device and generating a representative display of the user's interactions superimposed on the first image; and detecting at least one predetermined compound touch event generated by the user's interaction with the touch screen, including a first touch event component at a first location on the touch screen indicative of a location and a second sequenced touch event component at a second location on the touch screen indicative of an operative action to be performed within said normal mode of operation, and performing said operative action indicated by said second touch event component at said touch screen location indicated by said first touch event component ignoring the second location.
 2. A method according to claim 1 wherein said generating a representative display includes providing an indication, on a display device controlling the gazed-at screen, of at least one finger location on the touch screen based hand held device thereby allowing a user to generate touch screen events which control the display device's display output.
 3. A method according to claim 2 wherein said providing an indication occurs only responsive to a trigger event initiated by the user of the touch screen and wherein said trigger event comprises at least one touch screen event representing a touch gesture in which the user touches the touch screen.
 4. A method according to claim 3, where the touch gesture comprises placing two fingers on the touch screen at any location, followed by a drag of only one finger.
 5. A method according to claim 3, where the touch gesture comprises swiping two fingers in vertical movement in opposite directions.
 6. A method according to claim 3, where the touch gesture comprises dragging a single finger from a predetermined location on the touch screen.
 7. A method according to claim 3 and also comprising sniffing for the trigger event while passing all touch screen events, other than the trigger event, for touch screen event processing toward modification of the display output.
 8. A method according to claim 3 and also including, responsive to said trigger event, grabbing all touch events generated by the user on the touch screen during a time period, and for each of said touch events, providing a sensory cue for location of at least one finger upon the touch screen.
 9. A method according to claim 3 and also including, responsive to said trigger event, blocking all touch events generated by the user on the touch screen during a time period, for touch screen event processing toward modification of the display output.
 10. A method according to claim 9 wherein the time period terminates upon occurrence of a release event.
 11. A method according to claim 10 wherein said release event comprises at least one touch screen event representing a touch gesture in which the user touches the touch screen.
 12. A method according to claim 1, where the release event comprises a press of one of the CE device physical buttons.
 13. A method according to claim 1, where the release event comprises a touch gesture in which the user touches the touch screen.
 14. A method according to claim 13, where the touch gesture comprises lifting all fingers from the touch screen.
 15. A method according to claim 13, where the touch gesture comprises placing an additional finger on the touch screen in any location.
 16. A method according to claim 1, wherein generating a representative display comprises displaying a visual cursor overlay upon the normal display of the CE device in question which is visible on the external gazed at surface.
 17. A method according to claim 16, where at least one characteristic of the visual cursor provides a characterization of the touch surface of at least one finger with the touch screen.
 18. A method according to claim 17 wherein said characteristic of the visual cursor includes the shape thereof.
 19. A method according to claim 17 wherein said characteristic of the visual cursor includes the color thereof.
 20. A method according to claim 17 wherein said characterization of the touch surface includes the size thereof.
 21. A method according to claim 17 wherein said characterization of the touch surface includes the angular orientation thereof.
 22. A method according to claim 16, where the visual cursor is drawn by the apparatus used to render the remote display of the hand held device.
 23. A method according to claim 16, where the visual cursor is drawn using a display device which is additional to the display device rendering the main remote display of the hand held device.
 24. A method according to claim 16 where the CE device comprises a smart phone.
 25. A method according to claim 16 where the CE device comprises a media player.
 26. A method according to claim 1, where the trigger event comprises a key press.
 27. A method according to claim 1, wherein generating a representative display comprises generating an audio sound, sensible to the user, of the at least one characteristic of which indicates to the user the relative location of his finger on the touch screen.
 28. A method according to claim 27 wherein said characteristic includes at least one of: pitch, height and volume.
 29. A method according to claim 1, wherein generating a representative display comprises generating a haptic indication, sensible to the user, of the location of at least one finger on the touch screen.
 30. A method according to claim 1 wherein said location of at least one finger comprises locations of each of a plurality of fingers upon the touch screen.
 31. A method according to claim 1 wherein said generating a representative display includes providing an indication, on a display device controlling the gazed-at screen, of at least one finger location on the touch screen based hand held device thereby allowing a user to generate touch screen events which control the display device's display output.
 32. A method according to claim 31 wherein said providing an indication occurs only responsive to a trigger event initiated by the user of the touch screen.
 33. A method according to claim 32 wherein said trigger event comprises at least one touch screen event representing a touch gesture in which the user touches the touch screen.
 34. A method according to claim 33 and also comprising sniffing for the trigger event while passing all touch screen events, other than the trigger event, for touch screen event processing toward modification of the display output.
 35. A method according to claim 32 and also including, responsive to said trigger event, grabbing all touch events generated by the user on the touch screen during a time period, and for each of said touch events, providing a sensory cue for location of at least one finger upon the touch screen.
 36. A method according to claim 35 wherein the time period terminates upon occurrence of a release event.
 37. A method according to claim 36 wherein said release event comprises at least one touch screen event representing a touch gesture in which the user touches the touch screen.
 38. A method according to claim 32 and also including, responsive to said trigger event, blocking all touch events generated by the user on the touch screen during a time period, for touch screen event processing toward modification of the display output.
 39. A method according to claim 1 wherein said operative action comprises a selected one of a range of predetermined operative actions.
 40. A method according to claim 1 wherein said sensing includes detecting a gesture in which a plurality of fingers contact the touchscreen and, within a predetermined time interval, some of the plurality of fingers are removed from the touch screen whereas others of the plurality of fingers remain in contact with the touch screen.
 41. A method according to claim 1 wherein the representative display comprises a visual cue indicative of location of the user's interaction and wherein an initial position of the visual cue on the first image is exactly the initial position of the user's interaction on the touch screen, suitably scaled to take into account difference in size between the touch screen and the gazed-at screen.
 42. A method according to claim 41 wherein the visual cue moves in each dimension to exactly reproduce the user's motions over the touchscreen, scaled to take into account difference in size between the two screens.
 43. A method according to claim 1 wherein the representative display comprises a visual cue indicative of location of the user's interaction and wherein an initial position of the visual cue on the first image is predetermined irrespective of the initial position of the user's interaction on the touch screen.
 44. A method according to claim 43 wherein the visual cue moves in each dimension to exactly reproduce the user's motions over the touch screen, scaled by predetermined visual cue speed factors irrespective of difference in size between the two screens.
 45. A method according to claim 1 wherein said hovering mode of operation is implemented in an Android system.
 46. A method according to claim 45 wherein an underlying view receives an onHoverEvent if an appropriate Listener is registered with the Android system.
 47. A method according to claim 1 wherein said second, hovering, mode of operation is implemented on a Wi-Fi display.
 48. A method according to claim 1 wherein said first image is displayed remotely.
 49. A method according to claim 1 wherein said gazed-at screen comprises a television screen.
 50. A method according to claim 1 wherein software functionality residing in the hand-held device implements said second, hovering, mode of operation.
 51. A method for interacting with an image, the method being operative in conjunction with a gazed-at screen, on which a first image is provided which is based on another image produced by a touch screen touchscreen based hand held device, the method including: in a first, normal, mode of operation, sensing a user's interactions with the touch screen based hand held device and modifying a display on the touch screen based hand held device accordingly; in a second, hovering, mode of operation, sensing the user's interactions with the touch screen based hand held device and generating a representative display of the user's interactions superimposed on the first image; and upon detecting a predetermined compound touch event generated by the user's interaction with the touch screen, passing from one of said modes of operation to another, the compound touch event including a first event component at a first location on the touch screen indicating a location on the touch screen and a predetermined second sequenced event component at a second location on the touch screen which a user is capable of generating concurrently with the first event component, said predetermined second event being reserved to trigger passage between said modes of operation.
 52. A method according to claim 51 wherein an interface displayed on the hand held device is unaware of the second hovering mode of operation and interacts with the second hovering mode of operation as though it were the first mode of operation.
 53. A system for interacting with a touch screen based hand held device application, the system being operative in conjunction with an external digital display having a gazed-at display screen, on which a first image is provided which is based on another image produced by a touch screen based hand held device, the system interacting with a native interface operative, in a first, normal, default mode of operation of the hand held device, for sensing a user's interactions with the touch screen based hand held device and modifying a display on the touch screen based hand held device accordingly, the system comprising: virtual hovering apparatus operative, in a second, hovering, mode of operation of the hand held device, triggered by a trigger event and terminated by a release event, for sensing the user's interactions with the touch screen based hand held device and generating a representative display of the user's interactions superimposed on the first image; and for detecting at least one predetermined compound touch event generated by the user's interaction with the touch screen, including a first touch event component at a first location on the touch screen indicative of a touch screen location and a second sequenced touch event component at a second location on the touch screen indicative of an operative action to be performed within said normal mode of operation, and performing said operative action indicated by said second touch event component at said touch screen location indicated by said first touch event component ignoring the second location.
 54. A computer program product, comprising a computer usable medium having a computer readable program code embodied therein, said computer readable program code adapted to be executed to implement a method for interacting with a touch screen based hand held device application, the method being operative in conjunction with an external digital display having a gazed-at display screen, on which a first image is provided which is based on another image produced by a touch screen based hand held device, the method including: in a first, normal, default mode of operation, sensing a user's interactions with the touch screen based hand held device and modifying a display on the touch screen based hand held device accordingly; in a second, hovering, mode of operation, triggered by a trigger event and terminated by a release event: sensing the user's interactions with the touch screen based hand held device and generating a representative display of the user's interactions superimposed on the first image; and detecting at least one predetermined compound touch event generated by the user's interaction with the touch screen, including a first touch event component at a first location on the touch screen indicative of a touch screen location and a second sequenced touch event component at a second location on the touch screen indicative of an operative action to be performed within said normal mode of operation, and performing said operative action indicated by said second touch event component at said touch screen location indicated by said first touch event component ignoring the second location. 